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ABSTRACT 

The paper introduces a virtual collaborative learning setting called ‘Net Economy’, which we established as part of an 
international learning network of currently seven universities. Using the Community of Inquiry framework as guidance 
and Canonical Action Research (CAR) as the chosen research design, the discussion forum of the online course is 
assessed regarding its critical thinking value. We thereby measure critical thinking with the help of the according model 
provided by Newman et al. (1995), which differentiates 40 indicators of critical thinking from ten different categories. 
The calculated critical thinking ratios for the analyzed two discussion threads indicate a strong use of outside knowledge, 
intensive justification as well as critical assessment of posts by the students. But at the same time there are also weak 
spots, like manifold repetitions. Based on these results we derive changes for the next course cycle in order to improve 
the critical thinking of the students. 
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1. INTRODUCTION 

Learning with new media is becoming increasingly popular in times of MOOCs and a continuing growth in 
the use of mobile devices. Outside the traditional classroom, learning can spread to more diverse settings, 
contexts, and locations. As at the same time virtual collaboration is gaining significance in business, virtual 
collaborative learning (VLC) as a sophisticated type of eLearning setting gains growing attention in both 
research and higher education practice. The accompanying pedagogical and technological innovations 
thereby create new challenges for the instructors and also redefine their roles. 

Successful eLearning requires learning environments that motivate the students and facilitate meaningful 
and worthwhile learning. A suitable design and a well-structured learning process need to consider social, 
technical and didactic challenges as interdependent dimensions of a good learning experience 
(Garrison 2011). VCL-settings thereby aim at the skills that are required and considered indispensable for 
virtual collaboration, involving team members working from various international locations and with 
heterogeneous cultural and educational backgrounds. 

This paper introduces our own VCL learning scenario ‘Net Economy’ as part of a continuous 
build-and-evaluate loop, while our improvement efforts follow the iterative steps of canonical action research 
(CAR): diagnosis, action planning, intervention, evaluation, and reflection (Davison et al. 2004; Susman & 
Evered 1978). Action research is committed to “[...] the production of new knowledge through the seeking 
of solutions or improvements to ‘real-life’ problem situations.” (McKay & Marshall 2001) It thus serves 
both, research and practice by studying the real world while considering a particular theoretical framework. 

For our eLearning action research the Community of Inquiry (Col) as introduced by Garrison et al. (2000) 
serves as the required theoretical framework, consisting of the three interdependent elements of 1) social 
presence, 2) cognitive presence and 3) teaching presence. While social presence is “the ability of participants 
to identify with a group, communicate purposefully in a trusting environment, and develop personal and 
affective relationships progressively by way of projecting their individual personalities” 
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(Garrison 2011, p. 23), cognitive presence is “the extent to which learners are able to construct and confirm 
meaning through sustained reflection and discourse in a critical community of inquiry” (Garrison et al. 2001). 
Teaching presence, finally, is “the design, facilitation, and direction of cognitive and social processes for the 
purpose of realizing personally meaningful and educationally worthwhile learning outcomes” and thus 
merges all elements in a balanced and functional relationship. 

While in previous papers we developed solutions mainly for various problems of teaching and social 
presence, like e.g. the use of a dedicated social network instead of a learning management system, we have so 
far not explicitly addressed the eLearning elements of the setting with regard to the required cognitive 
presence. According to the Col framework students need to be “engaged in a collaborative and reflective 
process which includes understanding an issue or problem, searching for relevant information, connecting 
and integrating information, and actively confirming the understanding” (Garrison 2011, p. 94). An obvious 
instrument to do so seem to be discussion forums, which are therefore commonly used in eLearning. 
Computer-mediated communication tools, like discussion forums, provide ways for learners to interact. 
Research suggests that asynchronous online discussions are even superior with regard to critical thinking and 
knowledge co-construction due to the available time for reflection and more critical and constructive 
contributions when compared to synchronous face-to-face discussions (Wang et al. 2009). On the other hand, 
in practice, it usually stays unassessed and questionable whether the initiated discussions really provided for 
the intended critical and reflective interaction among the participants and thus for cognitive presence. 

In the following we, therefore, measure and analyze the critical thinking ratio of the discussion forum that 
we established in our ‘Net Economy’ setting as part of a required discussion task. Based on this diagnosis 
step we will then derive interventions that aim at improving the critical and reflective thinking and thus the 
cognitive presence. After a short introduction of our VCL-setting in section 2, we will outline the diagnosis 
approach as adapted from Newman et al. (1995) and assess the critical thinking in our implemented 
discussion forum in section 3. In section 4 we will then derive interventions as improvement suggestions for 
the next course cycle before we conclude in section 5. 


2. THE NET ECONOMY VCL-SETTING 

‘Net Economy’ is an annual cross-location virtual collaborative learning setting (VCL) in the framework of 
an international learning network which we established in 2008. The setting targets participants with 
heterogeneous educational backgrounds in the fields of business and economics, business informatics, 
international economy and hotel & tourism management, as well as different cultural backgrounds from 
Germany (Berlin, Miihlheim an der Ruhr, Soest), Indonesia (Jakarta), Crimea (Simferopol) and Sweden 
(Kristianstad). The course is taught in English and deals with entrepreneurship and business models in the net 
economy. Between 80 and 150 students took part in each course cycle so far and worked in virtual 
intercultural teams on exercises concerning eBusiness models and virtual collaboration. All partners agreed 
on the general learning scenario, leading to a stable VCL-concept which we systematically improve using the 
previously mentioned canonical action research approach. 

‘Net Economy’ is a complete online class with only three parallel classroom meetings at the various 
locations at the beginning of the different phases of the course, as depicted in fig. 1 . A closed social network 
(established with Wordpress and several plugins) serves as the major coordination platform for the course 
and facilitates social presence. Every student needs to set up a profile and join a team within the first weeks 
of the class, called Preparation Phase. During these weeks, the whole concept is introduced and the different 
tasks and roles within a team are described and assigned. The students team up to groups of five, in which 
every team member has a very specific role to fulfill. Only two members of the same university are allowed 
to be part of the same group so that a high diversity is given in each team. In the following Knowledge 
Development Phase, the students are then introduced to some major characteristics, concepts and challenges 
of eCommerce and the Web 2.0 and to major types of business models on the web. On each of the topics 
introduced by the instructors via eLecture, the individual students as well as the teams have to complete and 
submit assignments. Each participating location is responsible for one of these eLectures and an according 
online assignment. Knowledge Development Phase, ends with a multiple choice test on the topics covered in 
the eLectures. In the final Case Study Phase every team is then asked to apply the acquired knowledge and 
develop a business plan for an assigned business model type using the so-called business model canvas. This 
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Case Study Phase includes a peer review process in which each team receives feedback from at least two 
other teams. The top rated teams then present their business plan during a final live online meeting, in which 
all participants will vote for the top E-Venture of the class. 
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Figure 1. The Net Economy Learning Scenario 

In the last course cycle accomplished between October 2014 and February 2015, students were asked to 
contribute to an online discussion individually as a first assignment. They could choose between two 
discussion threads on which they got a reading and had to write a first forum post pointing out their personal 
opinion and thoughts regarding the topics of the articles. In addition, they had to comment on at least two 
other posts and emphasize in their statements whether they support or rebut the post they are answering to, in 
order to facilitate critical and reflective thinking. Discussion thread 1 discussed the online available WIRED 
article “The Web is dead. Long live the Internet.” by Chris Anderson and Michael Wolff and initiated 
discussion with the question; “How does the article and the discussion around it reflect today’s eCommerce 
trends and what consequences would you expect from the development for popular online businesses?” 
Discussion thread 2 dealt with the as well online available Fast Company article “Encyclopaedia Britannica 
(EB) is dead, long live Encyclopaedia Britannica”, raising the question: “Why did the original business 
model have to change and how did the Encyclopedia Britannica adapt? Do you believe in the viability of the 
business model?” The not any further moderated forum aimed to facilitate discourse and make the students 
think critically about the provided articles. 

In total 80 members were registered on the ‘Net Economy’ network in the last course cycle. Next to the 
lecturers 13 teams of five active students, in total 65 students, were working on the platform. 56 students 
(86%) participated in this first individual assignment and 166 replies were posted, which are about three 
posts per participating student, indicating that the students fulfilled their task of writing one statement and 
two replies. To investigate the qualitative value of the forum posts, however, a closer look has to be taken at 
the posts themselves and a more qualitative analysis is required, as accomplished in the following chapters. 


3. METHODOLOGY & APPLICATION OL METHOD 
3.1 Critical Thinking 

In the literature, many definitions of critical thinking can be found. According to the Foundation for Critical 
Thinking (2015) the word “critical” derives etymologically from two Greek roots: “kriticos” (meaning 
discerning judgment) and "kriterion" (meaning standards). Etymologically, then, the word implies the 
development of “discerning judgment based on standards.” Ennis (1993) defines critical thinking as 
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“reasonable reflective thinking focused on deciding what to believe or do.” In his opinion, a person needs to 
e.g. judge the credibility of sources and the quality of arguments, identify conclusions, reasons, and 
assumptions or develop and defend a position on an issue. Critical thinking therefore requires a wide set of 
information, generating processes and standards to base one’s thinking on and relies on the person involved 
to have the intellectual commitment to make use of these skills and standards to guide his/her behavior. 

Thus, critical thinking is a state of thinking that is beyond one’s own, or even groups, interests and is 
dependent on the quality of standards and depth of experience the thinker has in respect to a particular 
problem or question. It also depends on different values and cultures, as differences in culture can have 
different effects and interpretations of being critical, as “in some cultures, being critical may be interpreted as 
‘argumentative’ or ‘being critical of others’” (Woo & Wang 2009). Simplified one can say that critical 
thinking involves seeking information, analyzing alternatives, evaluating the alternatives in relation to your 
aims and requirements and reaching a conclusion to the problem or answer. 

In addition, critical thinking can also relate to other important areas in learning. Lai (2011) for example, 
says that “critical thinking skills relate to several other important student learning outcomes, such as 
metacognition, motivation, collaboration, and creativity.” Being able to assess one’s own arguments and 
reasoning is necessary for self-regulated learning. Tasks, which spark interest and are challenging the 
students usually call for critical thinking and trigger motivation. Collaboration requires the students to think 
in diverse perspectives and the more diverse they can think the better collaborators they may become. 
Opportunities for collaboration may also encourage higher quality thinking itself. Finally, also creative 
thinking incorporates attributes of critical thinking, such as open-mindedness and flexibility. All these 
attributes enhance the learning experience for the students and stimulate intellectual and personal growth 
(Lai 2011). 

3.2 Assessing Critical Thinking through Content Analysis 

In literature, many instruments are described which focus on content analysis and aim at providing evidence 
of learning and the knowledge construction taking place. These instruments represent a wide variety of 
approaches and differ in their level of detail, type of analysis categories and most important the diversity of 
their theoretical base (Wever et al. 2006). Content analysis instruments need to be objective, reliable, 
replicable, and systematic (Rourke et al. 2001). Instruments to measure critical thinking in terms of content 
analysis in a thematic unit of analysis were for example introduced by Newman et al. (1995) and Bullen 
(1998). 

Whereas Bullen's framework is based on different conceptualizations of critical thinking and consists of 
four different categories of critical thinking skills, Newman et al.'s approach is based on Garrison's (1991) 
five stages of critical thinking and Henri's (1992; cited in: Wever et al. 2006) cognitive skills. In his papers. 
Garrison describes critical thinking as a problem-solving process in which the critical thinker will move 
through five stages: problem identification, problem definition, problem exploration, problem applicability, 
problem integration. Marra et al. (2004) evaluate the Newman et al. protocol to be good for high-level 
descriptive data of what is happening in a discussion and clearly defined with numerous codes and coding 
rules. In our study we, therefore, used this approach to analyze the critical thinking of our students in the 
above-described discussion scenario. 

A prerequisite for this method, however, is that critical thinking indicators are identifiable, or in other 
words “critical thinking should be defined in such a way that it is measurable’’ (Woo & Wang 2009). In order 
to cover all possible types of posts in a discussion forum, Newman et al. (1995) expanded the five stages into 
the following 10 categories of critical thinking indicators: Relevance, Importance, Novelty, Ambiguity, 
Outside Knowledge, Linking ideas. Justification, Critical Assessment, Practical Utility and Width of 
Understanding. Each of the indicators has a pair of opposites, one for in-depth processing, and one for 
surface learning (Newman et al. 1995). Newman et al. have developed a full set of 40 indicators within the 
ten categories, as depicted in fig. 2. 
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Figure 2. Excerpt of coding indicators of the Newman et al. framework (Marra et al. 2004) 


While the categories are provided in the left column, the individual indicators and their pairs with the 
matching codes follow in the second and third column. A positive indicator for the Novelty category could, 
for example, be new problem-related information (NP-h) provided by a student or the welcoming of new 
ideas (NQ-h). Negative indicators would, for instance, be repetitions (NP-), or even the dismissal of new 
ideas of a previous speaker (NQ-) as well as irrelevant statements (R-). 

Along these provided categories with their various indicators, every post needs to be evaluated separately 
and codified accordingly. Statements may cover phrases, sentences, paragraphs or mere messages containing 
one unit of meaning and referring to least one of the indicators. It may even happen that some indicators 
overlap with each other (see figure 3 for example). This is why the coding needs to be carried out in 
chronological order, evaluating the first post first and the later posts to the end of the evaluation sequence. 
Only in this manner, the posts can be assessed properly and new content or repetitions can be identified 
correctly. Furthermore, the links between the different individual threads can be identified this way. But 
rather than classifying and coding every single word it is reasonable and even suggested to mark and count 
the obvious examples, and ignore the intermediate shades of gray (Newman et al. 1995). 

After assessing all posts in this way the approach allows for the calculation of a critical thinking ratio 
based on formula 1 depicted below. The x ratio with x identifying the category is calculated by subtracting 
the sum of all negative indicator from the sum of all positive indicators, divided by the sum off all indicators. 
The resulting measure thereby expresses solely the quality of the text without depending on the quantity of 
participants. This way ten critical thinking ratios are calculated, one for each category, and all of them in a 
range of —1 (all uncritical, all surface) to +1 (all critical, all surface) (Newman et al. 1995). 

x'^ — 

X ratio = r — 

Formula 1. Calculation of critical thinking ratio (simplified according to Newman et al. 1995) 

3.3 Forum Analysis 

The following analysis of the above-explained discussion forum was conducted with the help of the students 
of the ‘Net Economy’ course themselves. One student of each team was assigned the role of a research 
partner and as such had to mark the transcripts of the discussion thread that he did not participate in himself. 
The research partners were introduced to the job and the approach in a web conference, and, in addition, a 
short tutorial was provided on how the free coding software QDAminer Lite can be used to do the job. The 
transcripts of the forums were provided to the students together with the indicators as a predefined project in 
QDAminer Lite. Nine out of the 13 research partners submitted complete and acceptable coding projects with 
four students covering discussion thread 1 and five students covering thread 2. 

The analysis to follow is a merger of these nine complete coding projects provided by the students. Fig. 3 
shows how the coding of a post looks like in QDAminer with the positive and the negative indicators being 
depicted in the right. Building on to these individual codings we then proceeded by summing up all 
appearances of the different indicators and by calculating the critical thinking ratio for each category. While 
the two discussion threads were coded separately we later also merged the results, since both threads were set 
up and managed in the exact same way. This way we were able to calculate critical thinking ratios also for 
the discussion forum as a whole. 
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Figure 3. Coding of forum posts 

Several problems accompanying the chosen analysis approach and research design need to be considered. 
Next to the apple-and-pear problem because of possible variances between the different codings of the 
students, also very small numbers of appearances of specific indicators lead to the risk of over-interpretation. 
We, therefore, do not consider the critical thinking ratios of the Practical Utility and the Width of 
Understanding category in our analysis, as these indicators were used too rarely. Another limitation to keep 
in mind is that the chosen critical thinking model does not answer any specific question or solve any specific 
problem, but rather develops an overall rating for the quality of the interactions between the students and 
their discussion behavior. 


4. RESULTS & ACTION PLANNING 

After merging the coding projects, the results shown in table 1 can be calculated for the ten categories. We 
were able to evaluate eight of the ten categories when considering the sample sizes. Only for the Ambiguity 
category a negative result was calculated, indicating that the coding students found many confusing 
statements. All other ratios are positive with mostly rather small differences between the ratios for the 
discussion threads 1 and 2. As Newman et al. do not provide a scheme for interpreting the results except 
for-1 indicating all uncritical and surface posts and H-1 indicating all critical and deep contributions, there is a 
clear need to compare the derived ratios with the ratios of comparable settings. This actually matches our 
CAR approach of a continuous build-and-develop loop, since we will derive interventions in the next step 
that aim at improving the critical thinking in the implemented discussion forum, with the critical thinking 
ratios allowing us to assess and evaluate any achieved improvement. 


Table 1. Critical thinking ratios in the ‘Net Economy’ forum calculated after Newman et al. (1995) 





Topic 1 


Topic 2 


Combined 

Category 

Indicator 

Ratio 

Indicator 

Ratio 

Indicator 

Ratio 



+ 

- 


+ 

- 


+ 

- 


R+ 

Relevance 

118 

46 

0,44 

125 

85 

0,19 

243 

131 

0,30 

1+ 

Importance 

58 

53 

0,05 

34 

11 

0,51 

92 

64 

0,18 

N+ 

Novelty 

201 

156 

0,13 

158 

75 

0,36 

359 

231 

0,22 

A+ 

Ambiguity 

35 

72 

-0,35 

19 

42 

-0,38 

54 

114 

-0,36 

0+ 

Outside Knowledge 

189 

27 

0,75 

102 

14 

0,76 

291 

41 

0,75 

L+ 

Linking Ideas 

62 

41 

0,20 

29 

31 

-0,03 

91 

72 

0,12 

J+ 

Justification 

184 

20 

0,80 

137 

31 

0,63 

321 

51 

0,73 

C+ 

Critical Assessment 

55 

12 

0,64 

74 

6 

0,85 

129 

18 

0,76 

P+ 

Practical Utility * 

10 

0 

*(1) 

5 

5 

0 

15 

5 

* (0,50) 

w+ 

Width of Understanding * 

15 

2 

* (0,76) 

12 

7 

0,26 

27 

9 

* (0,50) 


* These categories are not considered any further due to too small samples. 


The high scores (CT>0,70) among the above results nevertheless point at several positive characteristics 
of the students discussion behavior. Outside knowledge was used a lot (CT=0,75), justifications were 
regularly provided (CT=0,73) and the posts were mostly critically assessed by fellow students (CT=0,76). A 
drill-down into one of the higher and one of the lower scores provides a clearer view of the assessed 
discussion. Table 2 adds an overview of the samples of single indicators from the two categories Linking 
Ideas and Outside Knowledge. Personal experience and previous knowledge, as well as additional course 
material and external resources, were introduced frequently by the students. Furthermore, they were also able 
to link facts and ideas, but thereby continuously repeated information and statements provided by other 
students before. 
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Table 2. Closer Look on the Outside Knowledge and Linking Ideas category and the according numbers 


Category 

+ Indicator 

Count 

Indicator 

Count 

0+ bringing outside 

OE-h 

Drawing on 

129 

OQ 

Squashing attempts to bring in 

6 

knowledge / 


personal experience 



outside knowledge 


experience to bear on 

OC-H 

Refer to course 

54 

0 

Sticking to prejudice or 

22 

problem 


material 



assumptions 


OM-h 

Use relevant outside 
material 

60 






OK-h 

Using previous 
knowledge 

41 





OP-H 

Course related 
problems brought in 

19 





OQ-h 

Welcoming outside 
knowledge 

1 




L+ linking ideas, 

L-h 

Linking facts, ideas 

81 

L- 

Repeating information without 

34 

interpretation 


and notions 



making inferences or offering an 
interpretation 



L-h 

Generating new 

10 

L- 

Stating that one shares the ideas 

38 



data from 



or opinions stated, without taking 




information 



these further or adding any 




collected 



personal comments. 



While these results indicate an overall positive discussion behavior from our point of view, the total 
number of posts points at a rather tightly focused fulfillment of the discussion task (one post + two replies). 
Students seem not to have participated in the discussion based on pure interest, curiosity or fun, but rather 
because it was a required task. The chosen topics and the way of raising and structuring the discussion seems 
not to have managed to turn the students’ extrinsic motivation into intrinsic motivation, which should have 
resulted in some sort of over-fulfillment of the discussion task. The identified numerous repetitions of 
information support this interpretation. 

As it is our goal to foster critical and reflective thinking and thus cognitive presence, we aim at engaging 
the students’ in a more intrinsically motivated discussion in the future. Considering the above results we 
derived the following three interventions regarding the implementation of the discussion forum for the 
upcoming course cycle: 1) More polarizing discussion starters will be developed, including other media 
types, like e.g. video statements, pictures or podcasts in order address the students media behavior more 
systematically. 2) The instructors will contribute to the discussion with prepared statements that open up new 
directions of thinking from time to time so that the students get new ideas and notice that their posts are not 
only counted but read. 3) Discussion forums with consistent requirements and settings will be used in several 
phases of the course and critical and reflective thinking as a desired learning outcome will be turned into a 
subject of discussion itself in order to evolve a discussion culture. 

After the next course cycle, we will assess the discussion forums and calculate the according critical 
thinking rations again in order to evaluate our interventions. This way we hope to have started an 
improvement process for the discussion forums as a major instrument of cognitive presence in eLearning. 


5. CONCLUSION, DISCUSSION & FUTURE WORK 

In this paper, we introduced the VCL-course ‘Net Economy’ and our iterative Canonical Action Research 
(CAR) approach aiming at a systematic improvement of the setting. Using the Community of Inquiry 
framework (Col) as guidance, we focused on cognitive presence as one major requirement of successful 
online learning, which we intended to provide for by implementing a discussion forum and an according 
discussion assignment among other tasks. We evaluated the discussion forum regarding its cognitive value by 
calculating the critical thinking ratios for all ten categories suggested by the Newman et al. (1995) critical 
thinking diagnosis model. Using this approach we were able to diagnose some strength and weaknesses of 
our discussion setting and derived interventions to be implemented in the upcoming course cycle. 

The Newman et al. model was simple to use and user-friendly since all categories and indicators are 
predefined. It provided for quality scores indicating the cognitive value of our discussion forum while not 
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depending on the number of participants. But despite this well-organized and user-friendly model, it turned 
out to be quite tedious and time-consuming to work through the assessed forum with regard to all 40 
indicators contained in the model. A problematic aspect is also the interpretation of the derived results. While 
it is clear that the more the values tend towards -1 or H -1 the worse respectively better discussion behavior 
they indicate, there is no guidance as where a good value might start or a bad value might end. 

Regarding the analyzed discussion forum we found that the students frequently introduced outside 
knowledge to the discussion, that they argued intensely, and that they thereby addressed posts critically. On 
the other hand, they often repeated information and posted a high number of rather confusing statements. 
However, the various cultural and educational backgrounds of the students might also play an important role 
in this matter, opening up the field for further research. From the calculated results, we finally derived 
interventions for the upcoming course cycle aiming at turning the students’ initial extrinsic discussion 
motivation into a more intrinsic one. A comparison of the results after the next course cycle will then allow 
us to assess the effects of the implemented changes, which would not be possible without a critical thinking 
diagnosis approach like the one provided by Newman et al. (1995) and used in this paper. Also, the process 
of analyzing a forum in such detail helped us a lot to understand the effects of the chosen discussion task with 
its strength and weaknesses better. 
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